A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences

Model-based active learning in hierarchical policies

Hierarchical task decompositions play an essential role in the design of complex simulation and decision systems, such as the ones that arise in video games. Game designers find it very natural to adopt a divide-and-conquer philosophy of specifying hierarchical policies, where decision modules can be constructed somewhat independently. The process of choosing the parameters of these modules manually is typically lengthy and tedious. The hierarchical reinforcement learning (HRL) field has produced elegant ways of decomposing policies and value functions using semi-Markov decision processes. However, there is still a lack of demonstrations in larger nonlinear systems with discrete and continuous variables. To narrow this gap between industrial practices and academic ideas, we address the problem of designing efficient algorithms to facilitate the deployment of HRL ideas in more realistic settings. In particular, we propose Bayesian active learning methods to learn the relevant aspects of either policies or value functions by focusing on the most relevant parts of the parameter and state spaces respectively. To demonstrate the scalability of our solution, we have applied it to The Open Racing Car Simulator (TORCS), a 3D game engine that implements complex vehicle dynamics. The environment is a large topological map roughly based on downtown Vancouver, British Columbia. Higher level abstract tasks are also learned in this process using a model-based extension of the MAXQ algorithm. Our solution demonstrates how HRL can be scaled to large applications with complex, discrete and continuous non-linear dynamics.Science, Faculty ofComputer Science, Department ofGraduat

Current state-of-the-art and gaps in platform trials: 10 things you should know, insights from EU-PEARL

Summary: Platform trials bring the promise of making clinical research more efficient and more patient centric. While their use has become more widespread, including their prominent role during the COVID-19 pandemic response, broader adoption of platform trials has been limited by the lack of experience and tools to navigate the critical upfront planning required to launch such collaborative studies. The European Union-Patient-cEntric clinicAl tRial pLatform (EU-PEARL) initiative has produced new methodologies to expand the use of platform trials with an overarching infrastructure and services embedded into Integrated Research Platforms (IRPs), in collaboration with patient representatives and through consultation with U.S. Food and Drug Administration and European Medicines Agency stakeholders. In this narrative review, we discuss the outlook for platform trials in Europe, including challenges related to infrastructure, design, adaptations, data sharing and regulation. Documents derived from the EU-PEARL project, alongside a literature search including PubMed and relevant grey literature (e.g., guidance from regulatory agencies and health technology agencies) were used as sources for a multi-stage collaborative process through which the 10 more important points based on lessons drawn from the EU-PEARL project were developed and summarised as guidance for the setup of platform trials. We conclude that early involvement of critical stakeholder such as regulatory agencies or patients are critical steps in the implementation and later acceptance of platform trials. Addressing these gaps will be critical for attaining the full potential of platform trials for patients. Funding: Innovative Medicines Initiative 2 Joint Undertaking with support from the European Union’s Horizon 2020 research and innovation programme and EFPIA